Combining Shape from Shading and Stereo: A Variational Approach for the Joint Estimation of Depth, Illumination and Albedo

Shape from shading (SfS) and stereo are two fundamentally different strategies for imagebased 3-D reconstruction. While approaches for SfS infer the depth solely from pixel intensities, methods for stereo are based on finding correspondences across images. In this paper we propose a joint variational method that combines the advantages of both strategies. By integrating recent stereo and SfS models into a single minimisation framework, we obtain an approach that exploits shading information to improve upon the reconstruction quality of robust stereo methods. To this end, we fuse a Lambertian SfS approach with a robust stereo model and supplement the resulting energy functional with a detailpreserving anisotropic second-order smoothness term. Moreover, we extend the novel model in such a way that it jointly estimates depth, albedo and illumination. This in turn makes the approach applicable to objects with non-uniform albedo as well as to scenes with unknown illumination. Experiments for synthetic and real-world images show the advantages of our combined approach: While the stereo part overcomes the albedo-depth ambiguity inherent to all SfS methods, the SfS part improves the degree of details of the reconstruction compared to pure stereo methods. An example of the reconstruction quality of our combined approach using only two views is given in Figure 1. As one can see, the reconstructed depth is quite detailed. Moreover, the computed illumination direction as well as the estimated albedo look reasonable.